A Typology of Student Engagement for American Colleges and Universities

The Carnegie classification system has served as a framework for research on colleges and universities for more than 30 years. Today, the system’s developers are exploring criteria that more effectively differentiate among institutions. One approach being considered is classifying institutions based on students’ educational experiences. This study explored whether it is possible to create a typology of institutions based on students’ experiences. Results indicated that such a typology could be created, and the types were somewhat independent of institutional mission (i.e., Carnegie classification). A Typology of Student Engagement 3 A Typology of Student Engagement for American Colleges and Universities Since its development in 1973, the Carnegie classification system has served as a framework for research on colleges and universities. These mission-oriented classifications were revised in 2000 to reflect recent changes in higher education, and the system’s developers continue to explore criteria that more effectively differentiate among institutions (McCormick, 2000). One approach being considered is classifying institutions based on students’ educational experiences (McCormick, personal communication, May 21, 2003). Examining students’ experiences is important because student engagement in educationally purposeful activities has desirable effects on student learning and success during college (Astin, 1977, 1993; Feldman & Newcomb, 1969; Kuh, Pace, & Vesper, 1997; Pace, 1990; Pascarella & Terenzini, 1991). Based on their review of 20 years of research, Pascarella and Terenzini (1991, p. 610) concluded, “one of the most inescapable and unequivocal conclusions we can make is that the impact of college is largely determined by the individual’s quality of effort and level of involvement in both academic and nonacademic activities.” Related Literature Student-engagement theory had its origin in the work of Astin (1984, 1985), Pace (1984), and Kuh and his colleagues (Kuh, Schuh, Whitt, & associates, 1991; Kuh, Whitt, & Strange, 1989). Although these writers used different terminology to describe their concepts of student engagement, their views were based on the simple, but powerful, premise that students learn from what they do in college. Research has strongly supported this assumption, indicating that engagement is positively related to objective and subjective measures of gains in general abilities and critical thinking (Endo & Harpel, 1982; Gellin, 2002; Kuh, Hu, & Vesper, 2000; Kuh & Vesper, 1997; Pascarella, Duby, Terenzini, & Iverson, 1983; Pascarella, et al., 1996; Pike, 1999, A Typology of Student Engagement 4 2000; Pike & Killian, 2001; Pike & Kuh, in press; Pike, Kuh, & Gonyea, 2003; Terenzini, Pascarella, & Blimling, 1996). Student engagement is also positively linked to grades (Astin, 1977, 1993; Indiana University Center for Postsecondary Research, 2002; Pike, Schroeder, & Berry, 1997) and persistence rates (Astin, 1985; Pike, Schroeder & Berry, 1997). A second important premise of the frameworks of Astin, Kuh, and Pace is that, even though the focus is on student engagement, institutional policies and practices influence levels of engagement on campus. For example, research has not been able to produce consistent relationships between students’ pre-college characteristics (e.g., gender, minority status and entering ability levels) and engagement during college (Bauer & Liang, 2003; Endo & Harpel, 1982; Hu & Kuh, 2002; Indiana University Center for Postsecondary Research, 2002; Iverson, Pascarella, & Terenzini, 1984; Kuh, Hu, & Vesper, 2000; Pike, 1999, 2000; Pike & Killian, 2001; Pike & Kuh, in press; Pike, Kuh, & Gonyea, 2003; Pike, Schroeder, & Berry, 1997). Moreover, the strength of those relationships, when present, was quite low. Studies by Pike and his colleagues have found that students’ background characteristics generally account for 1% to 5% of the variance in levels of engagement (Pike 1999, 2000; Pike & Killian, 2001; Pike & Kuh, in press, Pike, Kuh, & Gonyea, 2003). The influence of institutional characteristics on student engagement extends well beyond global characteristics such as size and institutional mission. Although both conventional wisdom and research studies suggest that attending small liberal arts colleges is associated with higher levels of engagement (Hu & Kuh, 2002; Kuh, 1981; Kuh & Siegel, 2000; Pascarella, Wolniak, Cruce, & Blaich, 2004), other studies have come to a different conclusion. For example, Pike, Kuh, and Gonyea (2003) found that differences in levels of engagement across Carnegie classifications disappeared after taking into account the background characteristics of the A Typology of Student Engagement 5 students. The most important institutional factors are thought to be the policies and practices adopted by institutions to increase student engagement. Several studies have shown that living on campus, as opposed to commuting to college, is positively related to engagement (Chickering, 1975; Pike & Kuh, in press; Terenzini, Pascarella, & Blimling, 1996). The gains associated with on-campus living are further enhanced by participating in learning communities, which substantially increases student engagement, self-reported gains in learning, and persistence (Indiana University Center for Postsecondary Research, 2002; Pike, 1999; Pike, Schroeder, & Berry, 1997; Zhao & Kuh, 2004). Given the powerful relationship between engagement and positive educational outcomes, it is not surprising that Astin (1985, p. 36) argued that “the effectiveness of any educational policy or practice is directly related to the capacity of that policy or practice to increase student involvement.” Some student engagement surveys are designed to assess the effectiveness of these institutional policies and practices (see Kuh, Pace, & Vesper, 1997). The most widely used instrument at this time is the National Survey of Student Engagement (NSSE) (Kuh, 2001a, 2003). The NSSE was developed as an alternative to reputationand resource-based ratings of news magazines and college guidebooks. Rather than ranking institutions, data from the NSSE survey, The College Student Report, provide colleges and universities with information about the activities in which their students engage and point to areas where improvement may be needed. Results from the first four years of the survey were generally consistent with previous theory and research on student engagement. Students attending small, selective liberal arts colleges tended to be somewhat more engaged than their counterparts at large public universities (Indiana University Center for Postsecondary Research, 2000, 2001, 2002). However, there was A Typology of Student Engagement 6 substantial variation in institutions’ student engagement scores within institutional categories as represented by Carnegie classification and size. Moreover, institutions with high engagement scores in one area generally did not have high scores in all areas (Kuh, 2001a; 2003). The findings that institutions with similar characteristics and missions differed substantially in both levels and types of engagement raise an important question: Is it possible to create a typology of engaging institutions that is independent of the traditional Carnegie classifications? The answer to this question is important for two reasons. First, as pointed out earlier, extant research shows that the Carnegie classification does not reliably distinguish institutions in terms of their educational effectiveness as represented by student engagement. Second, being able to classify institutions according to levels of student engagement will make it possible to identify colleges and universities that may be used as benchmarks by other schools with similar missions and other characteristics. For these reasons we set out to determine whether four-year colleges and universities could be sorted into meaningful categories based on patterns of student engagement. Research Methods A variety of approaches can be used to generate typologies (see Aldenderfer & Blashfield, 1984; Kruskal & Wish, 1978; Rummel, 1970). The approach selected for the present research was Q factor analysis (Burt 1937; Stephenson, 1953). According to Cattell (1952, p. 101), “Q technique is most useful if one wishes immediately to see how many types there are in a population and to divide it up into types.” One advantage of Q factor analysis over cluster analysis is that institutions can belong to more than one engagement type (Gorsuch, 1983). An important practical advantage of Q factor analysis, as opposed to multidimensional scaling, is A Typology of Student Engagement 7 that commercially available factor analysis programs can accommodate very large data sets (Kruskal & Wish, 1978). Participating Institutions The data for this study came from the 2001 administration of The College Student Report. The NSSE 2001 respondents included 177,103 first-year and senior students who were randomly selected from the populations of 321 participating colleges and universities. Students at 261 institutions had the option of responding either via a paper-and-pencil questionnaire or via the Web. Sixty schools opted for web-only administration of the survey. Of the 321 institutions, 4 were excluded from the study because of specialized missions and/or very low numbers of respondents. Table 1 displays the characteristics of the NSSE 2001 institutions and a national profile of all four-year colleges. These data show that the NSSE institutions were very similar to the national profile in terms of geographic region and urban-rural location. BaccalaureateGeneral colleges were underrepresented among the NSSE participants, whereas Doctoral/Research-Extensive and Baccalaureate Liberal Arts institutions were overrepresented among the NSSE participants. ______________________________ Insert Table 1 about here ______________________________ Only seniors were included in the current study for two reasons. First, they have had a wider range of experiences during college and arguably can provide more informed reports about a variety of college activities. Second, the experiences of first-year students and seniors differ substantially in terms of curriculum (coursework for first-year students emphasizes general education, while seniors are concentrated in the major) and out-of-class experiences (first-year A Typology of Student Engagement 8 students spend more time on formal extracurricular activities while seniors may have studied abroad, done internships, and so forth) (Pascarella & Terenzini, 1991). Thus, it would be difficult to construct a meaningful institutional typology of engagement from a combined sample of both first-year and senior students. The overall average unadjusted institutional response rate for NSSE 2001 seniors was 41.8%. Response rates ranged from 9.1% to 69.7%. About 69% of the seniors completed the paper version of the survey, and 31% completed the survey via Web. Generally, administration mode does not affect the results, with the exception that web respondents tend to report greater use of electronic technology (Carini, Hayek, Kuh, Kennedy, & Ouimet, 2003). Table 2 displays the characteristics of NSSE 2001 senior respondents in comparison to the characteristics of all seniors at the participating institutions. The results presented in Table 2 indicate that women tended to be overrepresented among the respondents, as were Caucasians and full-time students. However, the observed differences between respondents and the total population were relatively small. ______________________________ Insert Table 2 about here ______________________________ Measures of Student Engagement The NSSE College Student Report asks students to indicate the frequency with which they engage in activities that represent good educational practice and are related to positive learning outcomes (Kuh et al., 2001). Self-report data is widely used in research on college effects, and the validity and credibility of these data have been studied extensively (see Aaker, Kumar, & Day, 1998; Baird, 1976; Berdie, 1971; Bradburn & Sudman, 1988; Converse & Presser, 1989; A Typology of Student Engagement 9 Gershuny & Robinson, 1988; Pace, 1985; Pike, 1995; Pohlmann & Beggs, 1974; Turner & Martin, 1984; Wentland & Smith, 1993). Research shows that self-report measures are likely to be valid under five conditions: (1) the information requested is known to the respondents; (2) the questions are phrased clearly and unambiguously; (3) the questions refer to recent activities; (4) the respondents think the questions merit a serious and thoughtful response; and (5) answering the question does not threaten, embarrass, or violate the privacy of the respondent or encourage the respondent to respond in socially desirable ways (Kuh, 2001b, p. 4). Studies indicate that the College Student Report meets these five criteria and provides accurate and appropriate data about students’ levels of engagement (see Kuh et al., 2001). Fifty questions from the College Student Report were summed to create 12 engagement scales. These scales were calculated using procedures similar to those used to calculate the NSSE benchmarks, and the content of the scales paralleled the content of the benchmarks. For example, many of the items comprising the Course Challenge and Student Effort, Writing Experiences, and Higher-Order Thinking scales in this study were drawn from the Level of Academic Challenge benchmark. Items included in the Active Learning Experiences and Collaborative Learning Experiences scales were taken from the Active and Collaborative Learning benchmark, and the items comprising the Course-Related Interaction with Faculty and Out-of-Class Interaction with Faculty scales were taken from the Student-Faculty Interaction benchmark. Similarly, many of the items in the Varied Educational Experiences and Use of A Typology of Student Engagement 10 Information Technology scales were drawn from the Enriching Educational Experiences benchmark. The items included in the Support for Student Success and Interpersonal Environment scales came from the Supportive Campus Environment benchmark. The Experience with Diversity scale included items from the Active and Collaborative Learning and Enriching Educational Experiences benchmarks. Group-mean generalizability analyses (Cronbach, Gleser, Nanda, & Rajaratnam, 1972; Kane, Gilmore, & Crooks, 1976; Pike, 1994) revealed that dependable (i.e., Ep ≥ 0.70) institutional means could be calculated using as few as 50 respondents. Data Analysis The data analysis was conducted in two phases. In the first phase, Q factor analysis was used to classify colleges and universities into types of engaging institutions based on similarities in their student-engagement profiles. Initially, engagement scale means were calculated for all 317 colleges and universities in the study. The institutional means were then normalized (i.e., transformed to z-scores) to eliminate scaling differences across the 12 student-engagement scales and prevent a general species factor from confounding the results (see Cattell, 1952). The data matrix was then transposed so that the columns were the 317 institutions and the rows were the 12 student-engagement scales. Correlations among the 317 institutions were calculated. There is no agreement as to whether correlations or distance measures should be used in Q factor analysis. Correlations, unlike distances, are measures of pattern similarity and may group together institutions that have similar patterns on the 12 engagement scales, but very different means (see Guertin, 1971; Rummel, 1970). The advantage of using correlations is that the data are ipsatized after they are correlated, and “a normalized-ipsatized data matrix has almost all of any general factor eliminated from it” (Gorsuch, 1983, p. 316). A Typology of Student Engagement 11 A principal components analysis, using varimax rotation, was performed using the BMDP 4M program (Dixon, 1992). An analysis using principal axis factoring of a reduced correlation matrix with squared multiple correlations in the diagonal did not produce substantively different results because the squared multiple correlations were also extremely close to 1.00. The number of factors (i.e., types) extracted was determined using eigenvalues, a scree test, and the substantive interpretability of the factors or types. Correlations and standard regression coefficients for institutions’ normalized engagement scores and factor loadings were used in naming the student-engagement types. Institutions with factor loadings greater than or equal to 0.50 were classified as scoring high on a type, whereas institutions with loadings less than or equal to –0.50 were classified as scoring low on a type. Institutions with factor loadings less than 0.50, but greater than –0.50, were classified as scoring neither high not low on a type. Means on the normalized engagement scales were calculated for these groups and used as an additional check in naming the types, In the second phase of the data analysis, classifications of institutions as high, low, or neither high nor low were cross-tabulated with measures of institutional mission (i.e., Carnegie 2000 classifications). Chi-square tests were performed, and the contingent proportions in the tables were used to interpret relationships between the student-engagement types and Carnegie classifications. Results Student-Engagement Types Six factors, representing 80% of the variance in institutional means, were extracted and rotated to identify student-engagement types. It is significant that a dominant general factor did not emerge from the analyses. The first factor accounted for approximately 21.5% of the A Typology of Student Engagement 12 variance in institutional means, and the second factor explained 16.5% of the variance. Even the sixth factor explained a non-trivial 8.4% of the variance in institutional means. Names of the engagement types, eigenvalues, and squared multiple correlations, prior to factor rotation, are presented in Table 3. ______________________________ Insert Table 3 about here ______________________________ Table 4 presents the correlations and standard regression coefficients for institutions’ student-engagement means and their loadings on the six rotated factors. Also included are group means for institutions classified as high, low, or neither high nor low on the factors. An examination of the results in the first subtable reveals that the first factor is bipolar, representing two different student engagement types. Institutional means on the Experiences with Diverse Groups scale were positively correlated with the factor loadings, whereas Interpersonal Environment means were negatively correlated with the factor loadings (0.50 and –0.58, respectively). The corresponding betas were 0.56 and –0.36, respectively. Institutional means for writing, active learning, and student success also were negatively related to the factor loadings, but the strength of these relationships was substantially less than the strength of the relationship for the interpersonal environment. Thus, the first factor seems to distinguish between institutions characterized by diversity and institutions characterized by positive interpersonal relations. The institutions typed by this factor can be characterized as either diverse, but interpersonally fragmented or homogeneous and interpersonally cohesive. An examination of the group means for institutions classified as either high or low on the first factor supports this interpretation. Institutions with high positive loadings on the first factor had a A Typology of Student Engagement 13 positive mean on the diversity scale (0.70) and a negative mean on the interpersonal environment scale (–1.19). Institutions with large negative loadings on the first factor had a positive mean on the environment scale (0.66) and a negative mean on the diversity scale (–1.06). ______________________________ Insert Table 4 about here ______________________________ Two engagement scales, Out-of-Class Interaction with Faculty and Varied Educational Experiences, were positively correlated with loadings on the second factor (0.64 and 0.72, respectively). The standardized regression coefficients for the two scales also were positive and statistically significant (0.45 and 0.61, respectively). Course-Related Interaction with Faculty also was positively related to the second factor, although the strength of the relationship was much weaker than for the other two scales. Means on the Varied Experiences and Out-of-Class Interaction scales for those institutions with high loadings were large and positive (1.51 and 1.62, respectively), whereas institutions with low loadings had negative means on the Varied Experiences and Out-of-Class Interaction scales (–1.01 and –1.07, respectively). These institutions can best be described as intellectually stimulating colleges and universities. Three engagement scales, Experiences with Diverse Groups, Support for Student Success, and Interpersonal Environment, were positively correlated with loadings on the third factor (0.36, 0.43, and 0.43, respectively). Standardized regression coefficients were also positive (0.51, 0.43, and 0.37, respectively). Institutions with high loadings on the third factor had positive means for the diversity (0.60), support (0.73), and environment scales (0.74), whereas institutions with low factor loadings had negative scale means (–0.98, –0.76, and –0.53, respectively). These colleges and universities are interpersonally supportive institutions. A Typology of Student Engagement 14 The correlations, regression coefficients, and group means for the fourth factor suggested that the underlying construct for this factor was engagement through information technology. The correlation between Use of Information Technology scores and factor loadings was 0.59, and the standardized regression coefficient was 0.82. The information technology scale mean for institutions with high loadings was 0.85, and the mean for institutions with low loadings on the factor was –1.29. The institutions typed by this factor are high-tech, low-touch universities. The fifth factor represented institutions that were academically challenging and supportive. These institutions has high levels of academic challenge and student effort. The correlation between institutional means on the Course Challenge and Student Effort scale and loadings on the fifth factor was 0.58. The standardized regression coefficient was 0.71. Institutions with high loadings on the fifth factor had a mean course challenge score of 1.81, and institutions with low loadings had a mean score of –0.86. The results in the final subtable indicated that the sixth factor represented institutions with high levels of collaborative learning. The Collaborative Learning Experiences scale was positively correlated with loadings on the sixth factor (0.52), and the standardized regression coefficient was 0.80. The group mean for institutions with high loadings on the sixth factor was 1.47, whereas the mean for institutions with low loadings was –1.50. These institutions were labeled collaborative in this study. Engagement Types and Carnegie Classifications Chi-square tests of the relationships between engagement types and Carnegie classifications indicated that the different types of engaging institutions were related to differences in institutional missions; although the relationships generally did not support conventional wisdom that small, private liberal arts colleges have the highest levels of engagement. The relationships A Typology of Student Engagement 15 between engagement types and Carnegie classifications are presented in Table 5. Chi-square tests revealed that loadings on the first factor were significantly related to Carnegie classifications (χ = 109.43; df = 10; p < 0.001). An examination of the cross-tabulation of the first two engagement types in the first subtable and Carnegie classifications shows that diverse, but interpersonally fragmented, institutions tended to be Doctoral/Research-Extensive, and to a lesser extent Doctoral/Research-Intensive, universities. All other Carnegie classes were underrepresented in this engagement type. Conversely, interpersonally cohesive colleges and universities tended to be Masters I and II institutions. Follow-up analyses revealed that the institutions with positive interpersonal environments tended to be private (70.2%) and have enrollments of less than 3,000 students (49.6%). ______________________________ Insert Table 5 about here _______________________________ Chi-square tests also indicated that institutions’ loadings on the second factor were related to their Carnegie classifications (χ = 65.76; df = 10; p < 0.001). The contingent proportions in the second subtable indicated that intellectually stimulating colleges and universities tended to be smaller Baccalaureate-Liberal Arts institutions. Chi-square results indicated that there was not a significant relationship between Carnegie classifications and interpersonally supportive institutions (χ = 15.38; df = 10; p > 0.05). In contrast, there was a statistically significant relationship between Carnegie classifications and high-tech, low-touch institutions (χ = 52.65; df = 10; p < 0.001). Institutions characterized by high levels of engagement using information technology tended to be Doctoral/Research-Extensive universities. A Typology of Student Engagement 16 Chi-square tests also revealed that there was a statistically significant relationship between loadings on the fifth factor and Carnegie classifications (χ = 33.01; df = 10; p < 0.001). Specifically, academically challenging and supportive colleges and universities tended to be baccalaureate institutions, both Baccalaureate-Liberal Arts and Baccalaureate-General colleges. Most were private (73.9%) and 86.9% had enrollments of under 3,000 students. There was also a statistically significant relationship between institutions’ loadings on the final factor and their Carnegie classifications (χ = 32.30; df = 10; p < 0.001). Contingent proportions revealed that Masters I and, to a lesser extent, Doctoral/Research-Intensive institutions tended to have high levels of engagement in collaborative learning experiences. Furthermore, these collaborative colleges and universities tended to be public institutions (66.7%) and have enrollments ranging from 3,000 to 10,000 students. Limitations While results are generally consistent with the results reported by NSSE across the first few years of its surveys, only one year of data was analyzed in this study. If more institutions participating in other years were included, the results might differ in unknown ways. The institutional categories were derived only from the responses of seniors. If a similar analysis was done using first-year students, different factors may have resulted. Also, the NSSE survey is relatively short and, as a result, some potentially positive educationally purposeful activities are not represented, such as experiences in residence halls, the performing arts, and so on. If questions related to these activities were included, perhaps different results would emerge. Also, if multiple institutional characteristics were employed in the analysis, such as percentage of students in different majors or a measure of students’ socioeconomic status, different institutional types might be produced. Finally, while Q factor analysis is an accepted A Typology of Student Engagement 17 methodology for constructing typologies, other methods could produce different categories of institutions. Perhaps most important, Q factor analysis is a correlational procedure, and the chisquare analyses provided measures of association. Consequently, the findings of this study are descriptive and do not imply causal relationships. Despite these limitations, the results of the present research do have important implications for theory and practice.